Papers with model capacity expansion
Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have demonstrated considerable proficiency in general natural language processing tasks. |
| Approach: | They propose a parameter-efficient sparsity crafting method which crafts dense models into sparse models using the mixture-of-experts architecture. |
| Outcome: | The proposed method significantly reduces computational costs and GPU memory requirements, while maintaining the quality of approximation in function space. |